Papers with evaluation process
BotEval: Facilitating Interactive Human Evaluation (2024.acl-demos)
Copied to clipboard
| Challenge: | Using language models to perform complex interactive tasks is becoming more common with the rapid progress in natural language processing (NLP) models. |
| Approach: | They develop an evaluation toolkit that enables human-bot interactions as part of the evaluation process. |
| Outcome: | The evaluation toolkit enables human-bot interactions as part of the evaluation process, rather than making judgements for a static input. |
RAGthoven: A Configurable Toolkit for RAG-enabled LLM Experimentation (2025.coling-demos)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have significantly altered the landscape of Natural Language Processing (NLP), but their use as a baseline method has not been extensive. |
| Approach: | They propose a tool for automatic evaluation of RAG-based pipelines that provides a simple yet powerful abstraction. |
| Outcome: | The proposed tool provides an automatic evaluation of RAG-based pipelines. |
Style Over Substance: Evaluation Biases for Large Language Models (2025.coling-main)
Copied to clipboard
| Challenge: | Ranking the relative performance of large language models based on Elo ratings is gaining popularity . however, the extent to which humans and LLMs are capable evaluators remains uncertain . |
| Approach: | They propose to evaluate machine-generated text across multiple dimensions using the Elo rating system . they propose to use crowd-sourced and expert annotators to rank models based on Elo ratings . |
| Outcome: | The proposed method improves the quality of LLM-based evaluations, but there is no improvement in crowd-sourced evaluations. |
The Perils of Using Mechanical Turk to Evaluate Open-Ended Text Generation (2021.emnlp-main)
Copied to clipboard
| Challenge: | Recent research has focused on open-ended text generation tasks because they are difficult to evaluate automatically. |
| Approach: | They conduct a survey of 45 open-ended text generation papers to determine whether models are reproducible . they then run story evaluation experiments with AMT workers and English teachers . |
| Outcome: | The results show that AMT workers and English teachers perform better when shown model-generated output alongside human-generated references. |
A Relation Extraction Dataset for Knowledge Extraction from Web Tables (2022.coling-1)
Copied to clipboard
| Challenge: | Existing datasets with relational web-tables are either synthetic, or very small in size. |
| Approach: | They propose to annotate relational web-tables against a human-annotated dataset using crowd sourced annotators from MTurk. |
| Outcome: | The proposed dataset has 50x larger number of column pairs than the existing human-annotated benchmark. |
Continuously Steering LLMs Sensitivity to Contextual Knowledge with Proxy Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches to optimize Large Language Models (LLMs) for knowledge conflicts are inefficient or ineffective for large models and are not suitable for black-box models. |
| Approach: | They propose a framework that can continuously steer LLMs’ sensitivity to contextual knowledge at a lightweight cost. |
| Outcome: | The proposed framework can steer LLMs’ sensitivity to contextual knowledge continuously at a lightweight cost. |
The birth of Romanian BERT (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Large-scale pretrained language models are available in high-resource languages, in particular English, or as multilingual models that compromise performance on individual languages for coverage. |
| Approach: | They propose to use a Romanian transformer-based language model to pretrained a large text corpus to evaluate the model. |
| Outcome: | The proposed model is open-source and can be used in production. |
Multi-Layered Evaluation Using a Fusion of Metrics and LLMs as Judges in Open-Domain Question Answering (2025.coling-main)
Copied to clipboard
| Challenge: | Existing methods for comparing machine-generated answers with reference are not perfect in terms of accuracy or cost. |
| Approach: | They propose to summarize long answers and use shortened versions to improve evaluation . they propose a multi-layered evaluation methodology that integrates different metrics tailored to various scenarios . |
| Outcome: | The proposed method outperforms existing evaluation methods but is more cost-effective than existing methods. |
LongLeader: A Comprehensive Leaderboard for Large Language Models in Long-context Scenarios (2025.naacl-long)
Copied to clipboard
Pei Chen, Hongye Jin, Cheng-Che Lee, Rulin Shao, Jingfeng Yang, Mingyu Zhao, Zhaoyu Zhang, Qin Lu, Kaiwen Men, Ning Xie, Huasheng Li, Bing Yin, Han Li, Lingyun Wang
| Challenge: | LongLeader aims to assess different LLMs' long-context comprehension abilities . long-constext comprehension is a key bottleneck for many use cases . |
| Approach: | They propose a leaderboard to assess different LLMs' long-context comprehension abilities . they offer open-source access to the benchmarks and maintain a dedicated website . |
| Outcome: | The proposed model assesses different LLMs on selected benchmarks and provides open-source access to the benchmarks. |
Evaluation of HTR models without Ground Truth Material (2022.lrec-1)
Copied to clipboard
Phillip Benjamin Ströbel, Martin Volk, Simon Clematide, Raphael Schwitter, Tobias Hodel, David Schoch
| Challenge: | Optical Character Recognition (OCR) is a well-established technique for digitising historical printed collections in libraries and archives. |
| Approach: | They propose to use masked language models to evaluate handwritten text recognition models . they propose to introduce GT-free metrics to evaluate models to ensure best results . |
| Outcome: | The proposed model evaluations are based on lexicon-based and masked language models. |
Domain-Agnostic Adapter Architecture for Deception Detection: Extensive Evaluations with the DIFrauD Benchmark (2024.lrec-main)
Copied to clipboard
| Challenge: | Existing research focuses predominantly on specific fields, which results in the need for clarity on linguistic markers associated with deception. |
| Approach: | They propose a domain-independent fraud detection benchmark with 100,000 honest and misleading statements in seven domains and a parameter-efficient finetuning adapter to improve tuning methods. |
| Outcome: | The proposed adapter outperforms all competition on the DIFrauD benchmark and is able to predict the performance of the proposed model. |
A Closer Look into Using Large Language Models for Automatic Evaluation (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing studies explore the use of large language models to evaluate text quality, but they differ in some details of the evaluation process. |
| Approach: | They propose to use large language models to evaluate text quality by giving LLMs instructions to evaluate samples by giving them a rating. |
| Outcome: | The auto Chain-of-Thought (CoT) used in G-Eval does not always make it more aligned with human ratings. |
WordNet-Shp: Towards the Building of a Lexical Database for a Peruvian Minority Language (L18-1)
Copied to clipboard
| Challenge: | WordNet-like resources are lexical databases with highly relevance information and data that could be exploited in more complex computational linguistics research and applications. |
| Approach: | They propose to build a WordNet database for a low-resourced and indigenous language in Peru . they propose to use word2vec similarity to compare definition glosses in a dictionary with the content of a Spanish WordNet . |
| Outcome: | The proposed database is based on a bilingual dictionary written in Spanish and an automatic evaluation process using a manually annotated Gold Standard in Shipibo-Koniba. |
A Systematic Survey and Critical Review on Evaluating Large Language Models: Challenges, Limitations, and Recommendations (2024.emnlp-main)
Copied to clipboard
Md Tahmid Rahman Laskar, Sawsan Alqahtani, M Saiful Bari, Mizanur Rahman, Mohammad Abdullah Matin Khan, Haidar Khan, Israt Jahan, Amran Bhuiyan, Chee Wei Tan, Md Rizwan Parvez, Enamul Hoque, Shafiq Joty, Jimmy Huang
| Challenge: | Large Language Models (LLMs) have gained significant attention due to their capabilities in performing diverse tasks across domains. |
| Approach: | They review the primary challenges and limitations causing inconsistencies in evaluations . early models could generate coherent text but limited to simple tasks . |
| Outcome: | The proposed evaluations are reproducible, reliable, and robust. |
Graders Should Cheat: Privileged Information Enables Expert-Level Automated Evaluations (2025.emnlp-main)
Copied to clipboard
| Challenge: | a lack of trust in graders on graduate-level physics and Olympiad-level math makes them unreliable grader. |
| Approach: | They propose to use a grader LM to evaluate the candidate LMs. |
| Outcome: | The proposed approach outperforms human graders on *RewardBench* and human expert grader on Olympiad-level math problems. |
ExPerT: Effective and Explainable Evaluation of Personalized Long-Form Text Generation (2025.findings-acl)
Copied to clipboard
| Challenge: | Evaluating personalized text generated by large language models is challenging, as only the LLM user, i.e. prompt author, can reliably assess the output. |
| Approach: | They propose an explainable reference-based evaluation framework that leverages an LLM to extract atomic aspects and their evidences from the generated and reference texts, match the aspects, and evaluate their alignment based on content and writing style. |
| Outcome: | The proposed framework achieves a 7.2% improvement in alignment with human judgments compared to the state-of-the-art evaluation methods. |
LENS: A Learnable Evaluation Metric for Text Simplification (2023.acl-long)
Copied to clipboard
| Challenge: | Existing metrics for text simplification are based on unitary or outdated models, making them unsuitable for this approach. |
| Approach: | They present a learnable evaluation metric for text simplification using language models . they also introduce a human evaluation framework that rates simplifications from several models a list-wise manner . |
| Outcome: | The proposed model correlates much better with human judgment than existing metrics. |
SciNews: From Scholarly Complexities to Public Narratives – a Dataset for Scientific News Report Generation (2024.lrec-main)
Copied to clipboard
| Challenge: | Scientific news reports are a bridge between academic and scientific publications . however, the pursuit of automated news reports faces challenges due to the insufficient availability of parallel corpora. |
| Approach: | They propose to use a corpus of scientific news reports to facilitate this paradigm development . they highlight the divergences in readability and brevity between scientific news narratives and academic manuscripts . |
| Outcome: | The proposed corpus includes academic publications and scientific news reports across nine disciplines. |